Papers with bidirectional language models
Acquiring Bidirectionality via Large and Small Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Existing unidirectional language models are still used for token-level classification tasks, but they lack bidirectionality. |
| Approach: | They propose to use bidirectional language models to train a small backward LM and concatenate its representations to those of an existing LM for downstream tasks. |
| Outcome: | The proposed model improves performance by more than 10 points in token-classification tasks and in rare domains. |
Dissecting Contextual Word Embeddings: Architecture and Representation (D18-1)
Copied to clipboard
| Challenge: | Existing work on learning contextual representations has used LSTM-based biLMs, but there is no reason to believe this is effective. |
| Approach: | They propose to use pre-trained bidirectional language models to learn contextual word embeddings for four NLP tasks and to use them to study the effects of architecture on endtask accuracy. |
| Outcome: | The proposed models outperform word embeddings for four NLP tasks and all learn representations that vary with network depth. |
Consistent Bidirectional Language Modelling: Expressive Power and Representational Conciseness (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing bidirectional language models lack the ability to utilise future contexts and the pre-determined left-to-right generation order. |
| Approach: | They propose a class of bidirectional language models that are consistent by definition and can be efficiently used both for generation and scoring of sequences. |
| Outcome: | The proposed models are consistent by definition and can be efficiently used both for generation and scoring of sequences. |